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1.1 

Game theory and evolution 

Modern game theory goes back to a series of papers by the mathematician 
John von Neumann in the 1920s. This program started a completely new 
branch of social sciences and applied mathematics. This early work on game 
theory is summarized in the seminal book "The Theory of Games and Eco- 
nomic Behavior" by John von Neumann and Oskar Morgenstern [115]. Ini- 
tially, game theory was primarily focused on cooperative game theory, which 
analyzes optimal strategies assuming that individuals stick to previous agree- 
ments. In the 1950's, the focus shifted to non-cooperative games in which 
individuals act selfish to get the most out of an interaction. At that time, game 
theory had matured from a theoretical concept to a scientific field influencing 
political decision making, mainly in the context of the arms race of the cold 
war. 

The basic assumption was that individuals act rationally and take into ac- 
count that their interaction partners know that their decisions are rational and 
vice versa. Based on a common utility function that individuals maximize, 
the actions of others can be predicted and the optimal strategy can be chosen. 
However, the underlying assumption of rationality is often unrealistic. Even 
in simple interactions between two individuals A and B, it is difficult to imag- 
ine fully rational decision making, as this often leads to an infinite iteration: A 
thinks of B, who is thinking of A, who is thinking of B and so on. 

One way to avoid this situation in economy is the idea of bounded rational- 
ity [32,91]. If the cost of acquiring and processing information is taken into 
account, individuals can no longer be assumed to do a fully rational analysis 
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of the situation. Instead, their rationality is bounded and the way they make 
decisions is very different. In this case the prediction of the action of others 
is usually no longer possible. Nonetheless, bounded rationality turned out 
to be a crucial concept to overcome the problematic rationality assumption in 
economics. 

A completely different way to deal with this problem in biology has been 
proposed by John Maynard Smith and George Price in the 1970s [61,62]. They 
introduced the idea of evolutionary game theory. The utility function (which 
is usually the payoff from the game) is identified with the reproductive fitness. 
In this way, successful strategies spread in the population and less successful 
strategies diminish. Two important aspects differ from the traditional setting 
of classical game theory: (i) Evolutionary game theory always deals with pop- 
ulations of individuals instead of two (or few) players, (ii) In contrast to the 
static classical game theory analysis, evolutionary game theory provides a nat- 
ural way to introduce a dynamics into the system. 

There are two interpretations of evolutionary game dynamics. The first one 
is the traditional setting, in which strategies are encoded by the genome of 
individuals and successful types spread in the population due to their higher 
reproduction. Examples from biology include the competition of different bac- 
terial strains [58], cooperation in virus populations [113], or the cyclic domi- 
nance of mating strategies in lizards [94,95]. Biological reproduction selects 
successful strategies and does not require rational agents or other forms of 
cognitive abilities. 

The second interpretation is cultural evolution. In this setting, successful be- 
haviors are copied by other individuals through imitation. Successful strate- 
gies propagate through imitation and learning. Although individuals now 
have to make decisions, this is very different from the rational decisions in 
classical game theory. Instead of analyzing the situation in detail, the players 
just imitate those that are more successful. Such strategies are possible even 
with minimal cognitive premises. This approach is taken for comparisons be- 
tween predictions of evolutionary game theory and behavioral studies. 

More recently, ideas of evolutionary game theory have been reintroduced 
to economics, where they are tackled with great mathematical rigor [85]. 

1.2 

The replicator dynamics 

Traditionally, evolutionary game djmamics is described for very large, un- 
structured populations. In this case, a differential equation governs the evo- 
lution of the densities of the different strategies [50, 102, 118], 



Xi = Xi {jZi - {tz)) . 



(1.1) 
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Here, x/ is the fraction of type i in the population, tt/ is the fitness of this 
type and (zr) is the average payoff in the whole population. If the fitness of 
a type is above the average fitness in the population, its density will increase. 
If the fitness is below the average fitness, then the corresponding density will 
decrease. Each type i has a fixed strategy. If the fitness values 71/ are fixed, 
we speak of constant selection. In general, rci depends on the composition 
of the population, i.e. on the fractions of all other strategies Xy. Then, (zr) 
becomes quadratic in the fractions xy. Therefore, the dynamics is nonlinear in 
general. Since the variables in the replicator equation represent the fractions of 
each strategy in the population, the natural coordinate system is a probability 
simplex, i.e., for 2 strategies, we have a line, for 3 strategies an equilateral 
triangle, for 4 strategies an equilateral tetrahedron and so on. 

As an example, let us consider the simplest possible games. These are so 
called two player normal form games or 2 x 2 games. Such games can be 
described by a payoff matrix of the following form 



A B 
A f a b 
B [ c d 



(1.2) 



This is a way to specify the interactions of two types: If A interacts with an- 
other A, it obtains a and h if it interacts with B. Similarly, B obtains c from 
interactions with A and d from interactions with B. The payoffs are deter- 
mined by the fraction of interactions with a given type. Since we have only 
two types, the population state is fully determined by x = Xi = 1 — X2- The 
payoffs are then tt^ = a x + h{l — x) and ZTg = c x + d{l — x). This leads to 
the replicator equation 

X = x(l - x)[{a -h - c + d)x + h - d] . (1.3) 

Apart from the trivial fixed points x = and x = 1, the replicator equation 
can have a third fixed point x* ior a > c and d > b or for a < c and d <h, 

^ d — h 

X* -. (1.4) 

a-h-c+d 

We can distinguish four generic cases [77], see Fig. |l.l[ 

• Dominance. In this case, one strategy is always a better choice, regard- 
less of the action of the opponent. Either A dominates B {a > c and 
b > d) or B dominates A {a < c and b < d). In the first case, the fixed 
point at X = 1 is stable and the fixed point at x = is unstable and vice 
versa in the latter case. 



• Bistability. This is the case for a > c and d > b. The fixed points at 
X = and x = 1 are stable, separated by an unstable fixed point x*. 
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Dominance Coexistence 




Fig. 1.1 The four dynamical scenarios of evolutionary 2x2 games. The arrows indicate 
the direction of selection, filled circles are stable fixed points and open circles unstable fixed 
points. In the neutral case, the entire line consists of neutrally stable fixed points. 



The corresponding games are called coordination games. What is the 
best strategy in such a game? As a first approximation, one can ask for 
the maximum payoff in the equilibrium. However, if the opponent is 
unreliable, one should also try to avoid large losses. This leads to the 
concept of risk dominance: The strategy that has a larger basin of attrac- 
tion is called risk dominant. In our case, strategy A is risk dominant for 
a-\-b > c-\- d (or, equivalently x* < \). For a + b < c + d (or x* > \), 
strategy B is risk dominant. 

• Coexistence. For a < c and h > d, there is a stable fixed point at x*. 
Hence, the population becomes a stable mixture of A and B types. Both 
X = and x = 1 are unstable fixed points. 

• Neutrality. For a = c and h = d, the replicator dynamics predicts neu- 
trally stable fixed points for all values of x. While this non-generic case is 
of limited interest in the replicator dynamics, neutral selection becomes 
an important reference case in the stochastic evolutionary djmamics of 
finite populations. 

In a genetic setting, the replicator equation is obtained when individuals 
reproduce at a rate proportional to their fitness. In cultural settings, the repli- 
cator equation results from individuals imitating better performing actions 
with a probability proportional to the expected increase in the payoff. More- 
over, the replicator equations are intimately connected to the Lotka-Voltera 
equations describing predator-prey dynamics in theoretical ecology. In fact, a 
direct mathematical mapping from one framework to the other exists [50]. 

In general, replicator equations can exhibit very rich dynamics. However, 
in most cases, the location and stability of fixed points can be determined an- 
alytically. In addition, symmetry properties can lead to constants of motion. 
In this case, the djmamics follows closed orbits. In higher dimensions, the 
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replicator equations can also exhibit deterministic chaos [87,88,90]. However, 
the replicator dynamics is just one particular variant of a deterministic selec- 
tion dynamics. If individuals switch to better strategies independent of the 
increase in the payoff, one obtains a dynamics sometimes called "imitate the 
better''. Naturally, this would change the form of the differential equation 
The replicator equation and "imitate the better" are two examples of 
non-innovative selection dynamics, i.e. a strategy that goes extinct will never 
re-appear. However, in social models, individuals could also choose the "best 
reply" to the current composition of the population, even if such a strategy is 
not present in the population. This is an example of innovative game dynam- 
ics for which equally simple differential equations as can no longer be 
defined. 

1.3 

Evolutionary games in finite populations 

While replicator equations have very nice mathematical properties and pro- 
vide many important insights into evolutionary game dynamics, they essen- 
tially describe the deterministic dynamics in an infinitely large population. In 
general, it is not a priori clear under which circumstances this is a good ap- 
proximation of the dynamics in a realistic system which is finite and subject 
to fluctuations. One important source of fluctuations is internal noise aris- 
ing from the finiteness of the population. To answer these questions, we take 
the following approach: First, we describe evolutionary game dynamics as 
a stochastic process in a finite population. Then, we increase the population 
size and infer under which circumstances we recover the replicator dynamics. 
This also allows us to identify parameter combinations for which it is a good 
approximation. Moreover, we can investigate whether other limits result in 
qualitatively different dynamics. 

1.3.1 

Stochastic evolutionary game dynamics 

Evolutionary games in finite populations have been considered for a long time 
in various fields such as theoretical ecology, behavioral economics or sociol- 
ogy. For computer simulations of multi-agent systems, this is the only natural 
approach to model the dynamics. In all cases, we have to specify the mi- 
croscopic mechanisms which govern the transmission of strategies from one 
individual to another. Examples for such selection dynamics are 

• Pairwise comparison processes. In this class of processes, two individu- 
als, a focal individual and a role model, are sampled at random from the 
population. The focal individual accepts the strategy of the role model 
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with probability p, depending on a payoff comparison. If both individ- 
uals have the same payoff, the focal individual randomizes between the 
two strategies. For example, the probability p could be a linear function 
of the payoff difference [104], 

1 TCf - TXy 

+ (1.5) 

Here, w {Q < w < 1) is the intensity of selection, which specifies the 
importance of neutral drift compared to the selection djmamics. This 
determines the noise intensity. The functions rCf and TCy are the payoffs 
of the focal individual and the role model, respectively, and Att is the 
maximum payoff difference. For w <^1, one speaks of "weak selection". 
Most importantly, the focal individual does not always switch to the bet- 
ter strategy - sometimes, it also adopts worse strategies. 

One common choice of a nonlinear function of the payoff difference for 
p is the Fermi function from statistical mechanics, leading to 



(1.6) 



Here, the intensity of selection relates to an inverse temperature and can 
be any positive number. For weak selection w <^ 1, the probability p 
reduces to a linear function of the payoff difference. For strong selection, 
w ^ CO, this process converges to the imitation dynamics. In this case, 
p becomes a step function being positive for rCf > TZf and negative for 
Ttr < TCf. In this limit, only the order of the payoffs is important - an 
arbitrarily small difference in the payoffs leads to the same outcome. 

Moran process. The Moran process is a classical model of population 
genetics [68] and has been transferred to game theory only recently 
[74, 101]. Here, one individual is chosen at random, but proportional 
to fitness. This individual produces one identical offspring. To keep 
the number of individuals constant, a randomly chosen individual is re- 
moved from the population before the offspring is added. The Moran 
process represents a simple birth-death process. To illustrate the selec- 
tion mechanism, it is instructive to think of a roulette wheel in which 
the size of the different fields is proportional to the fitness. Fitness is 
a convex combination of a background fitness (usually set to one) and 
the payoff from the game, i.e. / = 1 — w -\- wn. Again, w determines 
the intensity of selection. For w = 0, selection is neutral and we have 
an undirected random walk. For w ^ 1, fitness equals payoff. How- 
ever, if the payoffs n can become negative, there is a maximum intensity 
of selection, because the fitness has to be positive. This restriction can 
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be overcome if fitness is defined as an exponential function of payoff, 
/ = exp [wn] . In this case, the selection intensity w can take any positive 
value [111]. 

• Wright-Fisher process. The Wright Fisher process has its roots in pop- 
ulation genetics as well. In contrast to the selection dynamics in the 
Moran process, where only one individual reproduces at a time, the 
Wright-Fisher process reflects discrete generations. In every generation, 
each of the N individuals produce a large number of offspring, propor- 
tional to their fitness. From this large offspring pool, a new generation 
again of size N is randomly sampled. Thus, the composition of the pop- 
ulation can change much faster. In principle, the population could go 
back to a single ancestor in a single generation. This reflects the fact that 
the Wright-Fisher process is not a simple birth-death process, but a more 
general Markov process. Although it becomes very similar to the Moran 
process under weak selection, the fixation probabilities (see next section) 
cannot be calculated exactly [54]. 

The replicator equation determines how the frequencies of strategies in the 
population change: For each point in state space, the direction of selection 
and its velocity is determined. For finite populations, we have to deal with 
probabilities instead. Moreover, the state space is discrete. In this discretized 
space, we can calculate into which direction the system will evolve with what 
probability. This imposes significant restrictions on analytically accessible re- 
sults. However, the general properties can already be inferred from the sim- 
plest possible case, consisting of only two strategies. 

1.3.2 

Fixation probabilities 

For non-innovative game dynamics, we have two absorbing states: either all 
individuals are of type A or of type B. One important determinant of the 
dynamics is given by the probabilities of fixation: If a mutation leads to a new 
strategy, how likely is it that this individual takes over the entire population? 

For the sake of simplicity, let us focus on birth-death processes in which 
only a single individual reproduces at a time (we thus exclude processes as the 
Wright-Fisher process). We consider a population of size N where the number 
of A players is ; and the number of B players is N — The probability to 
increase the number of A individuals from ; to ; + 1 is Tj^. Similarly, is the 
probability to decrease ; by 1. Our goal is to calculate the fixation probabilities 
(pj, i.e. the probability that ; individuals of type A succeed and take over the 
population. For the absorbing states, we have 



^0 = and = 1. 



(1.7) 
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For the intermediate states, the fixation probabilities are given by 

cpj = Tr4>j_, + {i-Tr-T+ )<Pj + T+<Pj+^ . (1.8) 

We can rearrange this to 

= -T^r {j>. - cpj_^ +7^+ . (1.9) 

Vj Vj+i 

This equation can be written as a recursion for the differences between fixation 

Tr 

probabilities, yy+i = 7yi/y, where 7y = Using this iteration, we find 

yi = (pi-(po = (pi (1-10) 

1/2 = (p2 - (pi = 7i(pi (1-11) 



k-l 

Vk = (pk- (pk-1 = n (1-12) 



N-l 
7=1 

As usual, the empty product is one, ny=i 7y = 1- Let us now calculate the sum 
over all yy. This sum is a "telescope-sum'' and simplifies to 

N 

J^yk = (pl- Jp0^+(p2 -(pl+(p3-(p2 + + (pN -(pN-1 = 1- (1-14) 

Using Eqs. {i.l2\ and ( |1.14| , we can finally calculate (pi, 

N N k-l ( N-l k \ 

1 = Ey^ = E^inTy = </>i 1+ EnTy (1-15) 

k=l k=l 7=1 V k=l 7=1 / 

Hence, the fixation probability of a single A individual, (pi, is given by 
1 

i + ErJnj=i77 



For = T^+, we have 7y = 1. Hence, all products are simply one and we find 
(pi = 1/N. This is the case of neutral selection, where all individuals have the 
same fitness. Any random walk in which the probability to move to the left or 
to the right is identical for the transient states leads to the same result. 
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So far, we have focused on the fixation probability of a single mutant in a 
resident population, (pi. In general, the fixation probability (pi is given by 



cpi = LVk (1-17) 

k=l 

= ^ifi+EriT/) (1.19) 

V k=ij=i ) 



i+EtVn;.i7; 



(1.20) 



For neutral selection, we have = Tr, which results in 7y = 1. In this case, 
the fixation probability reduces to (^i = z/N. 

Formally, the fixation probability can be calculated in systems in which the 
replicator equation predicts coexistence, i.e. no fixation. However, it can also 
be shown that in these cases, the average time until fixation grows exponen- 
tially with increasing population size [3] and increasing intensity of selec- 
tion [109]. 

Often, the comparison between the probability that a single A individual 
takes over a population of N — 1 resident B individuals, pA — ^rid the 
probability that a single B individual takes over a population of N — 1 resident 
A individuals, p^, is of interest in order to determine in which state the system 
spends more time [71]. Formally, the probability p^ is equal to the probability 
that N — 1 individuals of type A fail to take over a population in which there 
is just a single B individual. Hence, we find 

Pb = 1 - ^N-l 

^ i + Eri'n}=i7; 
i + E£i^n;.i7; 
^ i + E£-/n^i7; i + Etfn^i7; 
i+E£-i^n;.i7; i+E£i^n;.i7; 

i+Eri'n^i7; 

N-l 



(1.21) 
(1.22) 

(1.23) 

(1.24) 

(1.25) 
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Therefore, the ratio of the two fixation probabilities is given by ^ = Hyli 7j- 
If this product is smaller than 1, we have pB < PAr if it is larger than 1, we have 
Pb > Pa- ^oy small mutation rates, pB < Pa n\eans that the system spends 
more time in the A state, because less invasion attempts are necessary to reach 
fixation by A. The A mutants have a higher probability to reach fixation in a 
B population compared to B mutants in an A population. 

1.3.3 

Fixation times 

Another quantity of interest in evolutionary dynamics of finite population is 
the average time until fixation occurs [24]. For two strategies, three different 
fixation times are of interest: 

(i) The average time tj until either one of the two absorbing states, A or B, 
is reached when starting from state This is the unconditional fixation 
time. For weak selection, this time increases with the distance between ; 
and the two absorbing boundaries. 

(ii) The conditional fixation time specifies the average time it takes to 
reach the absorbing state A when starting from state provided that A 
is ultimately reached. The time increases with the distance between 

the states ; and A. If fixation of strategy A is almost certain, is very 

similar to the unconditional fixation time t. Of particular interest is f^, 
or t"^ for short, which denotes the average fixation time of a single A 
mutant in a resident B population. 

(ii) In analogy to t^, tj represents the average time to reach the absorbing 
state B when starting in state ; (i.e. with ; individuals of type A and 
N — ; of type B), provided that B is ultimately reached, tj increases with 
the distance between state ; and B. 

Unconditional fixation time. The unconditional average fixation time tj, start- 
ing from state is determined by 



This equation can be rationalized as follows: In one time step (this results in 
the 1), the process can either move to ; — 1, stay in ; or move to ; + 1. Then, 
the fixation time from that state matters. When we start in or in N, fixation 
has already occurred, thus = = 0- Eq. \1.26\ can be written as 



t 



j-i + {l-Tr-Tptj + T^tj^,. 



(1.26) 



1 



(1.27) 
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Tr 

With the notation jj = ^ from above, iteration yields 

zi = h-to = h (1-28) 
Z2 = t2-h = jih-^ (1.29) 

Z3 = t3-h = 7271^1 - ^ - Tj^If (1-30) 



k-1 k-1 2 

= h- = ^1 n - E ^ n (1-31) 

m=l /=1 ^/ m=/+l 

For the sum of the z^, we find 

N 

E ^/c = ^7+1 ~ ^7 + ^7+2 - ^7+1 + • • • + -^N-l = -ij (1.32) 



In particular, we have for ; = 1 



N N-1 k N-1 k 1 

tl = -Ez;c = -*l E nTm+ E n 7m. (1.33) 



fc=2 fc=l m=l k=l 1=1 m=/+l 



From this, the first fixation time, fi, is obtained 



^ N-1 k Y k 



= l + r^-irf ^ ^ ^ f+ n 7;- (1-34) 

1 + Lfc=i 1 ly=i 7/ k=i 1=1 h 7=/+i 

^ V ^ 

Here, (pi is the fixation probability given by Eq. ( |1.16| ). Thus, the average un- 
conditional fixation time for general ; is finally given by 

N N-1 k N-1 k 1 k 

^7 = - E = E n 7m + E E n 7m. (1.35) 

k=j-\-l k=j m=l k=j 1=1 m=/+l 

It is important to notice that the variance of these fixation times is usually 
high, depending on the population size, the game and the intensity of selec- 
tion [20,109]. In particular for coexistence games, where the replicator dy- 
namics predicts a stable coexistence of A and B, the fixation times do not only 
diverge with the population size and the intensity of selection, but they repre- 
sent averages of a very broad distribution [109]. 
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Conditional fixation times. Given that the process reaches the absorbing state 
with A individuals only, how long does this take when starting in state ;? To 
calculate this time f^, we follow Antal and Scheuring [3]. It is convenient to 
start from 

cpjtf = cPj.tTr{tf_, + l)+cpj{l -T7- T+){tf + 1) + <Pj+^T+ {tf^^ + 1). 

Here, is the fixation probability of ; individuals of type A, see Eq. ( |1.2Q| ). 
With the abbreviation = (pjfj^, we can write this as 

ef,r-ef^ef-ef_J^-^. (i.36) 



Eq. \136) has the same structure as Eq. ( |1.27| ). Thus, we can use a similar 
iteration as above to obtain 

= n-rm-Elr n Tm. (1.37) 

m=l 1=1 m=/+l 



At the lower boundary, we have 9^ = 0, because (p^ = 0. We also have 
= at the upper boundary, because = Summing over W]^ leads to 
EfcLy+i ^fc = ~^f' particular, for ; = 1, we obtain 

N-l k ^ k 

^i-LLpf n ym. (1.38) 

k=l 1=1 m=/+l 

Often, this is the quantity of interest, because it corresponds to the average 
time it takes for a single mutation to reach fixation in the population. For 
general we have 

^ N-l k N-l ^ 1 k 

*f - -^f^ E n 7. + E E n 7.. (1.39) 

k=j m=l k=i 1=1 r] J- 1 m=l+l 

For (pi = (pj = 1 (certain fixation of A), reduces to the unconditional fixation 
time, Eq. {135\ . 

For completeness, let us also calculate the average time t? until type B 
reaches fixation in the population. Instead of Eq. ( |1.36| ), we now have 



^j-i 




(1.40) 
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where = 1 — (pj is the fixation probability for reaching state B and 0? = 4^jtj. 
Again, we have = = 0. Now, we start our iteration from ; = N — 1, 

VN = ^N-^N-l = -4-l (1-41) 

VN-1 = ^^i-^^2 = -^Li-^ + |^ (1-42) 



m=l f^^-^ 1=1 ^N-l m=/+l 



Summation yields E^]v-y '^N-k — ^f- From; = N — 1, we find for the fixation 
time of a single B mutant 

N-l k z k 1 

tl-^-LLp^ n (1-43) 

k=l 1=1 ^N-l m=l+l f^-m 

Of course, this quantity can also obtained from Eq. \13S) by symmetry argu- 
ments. Finally, we obtain for the time until fixation of B when starting from 
an arbitrary number of B individuals 

X N-l k 1 N-l k z -1 k 1 

'f=-«-i^ E E E^5^ n -^(144) 

r] k=N-jm=l rN-m k=N-jl=l r] ^N-lm=M '^'^^^ 



-m 



This formulation is valid for general birth-death processes. In addition to the 
fixation probabilities, the two types of fixation times are of particular interest 
to characterize the evolutionary process because they represent global quanti- 
ties that include information on all transition probabilities. 
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1.3.4 

The Moran process and weak selection 

As a specific example, let us consider the frequency dependent Moran process 
[74,101]. Apart from calculating the above quantities for a specific case, our 
goal is to find simpler expressions for the fixation probabilities under weak 
selection. In this case, we are close to neutral selection, which seems to be a 
biologically relevant limit [17,78]. 

First, we need to specify the payoffs of the two types A and B. The payoffs 
of A and B individuals, tca and ZTg, are given by 

/ - 1 N -j 
= + (1-45) 

ns - ^c+^^rf- (1-46) 

Here, we have excluded self interactions, i.e. in a population of ; individuals 
of type A, each one of them interacts with j — 1 others of its type. Fitness is 
assumed to be a linear combination of background fitness (which we set to 1) 
and the payoff, 

/a = l-W + WTlA (1.47) 
fs = l-W + W7tB. (1.48) 

The transition probabilities are 

r+ = ^ (149) 

_ (N-/)/b / , - 

I ~ ifA + {N-j)fBN- ^'-^^^ 

For the ratio of the transition probabilities, we have 

7- = ^ = ^ = ^~^ + ^^^ (1 51) 

T+ /a + wua ' 

Let us now consider the limit of weak selection, i.e. <C 1, and derive an 
approximation for the fixation probability (pi . For weak selection, jj simplifies 
to 

The product in Eq. l |1.16^ can then be simplified to 

k k k 

HT; « YV^-^i^A-TtB))^l-wY,{nA-nB). (1.53) 

/=1 ;=1 /=1 
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Next, we introduce a new notation for zr^ — zTg. From Eqs. \1A5\ and ( |1.46| ), 
we find 



a — h — c + d 
N-1 



-a + hN-dN + d 
N-1 



(1.54) 



With this we can solve the sum over the payoff difference. 



ti^A - nB) = E(« + ^) = "^^^ + = + (f + ^) ^- (1-55) 

/=i /=i 



Now we have derived a simple form for ny=i 7/ urider weak selection. Let us 
analyze the ratio of fixation probabilities: 



PB 



N-1 N-1 

= n '>'/ ~ 1 - ^ E - 

^'^ /=1 ;=1 



(1.56) 



1 — w 



w 



|(N-l) + | + p (N-1) 



1 - ^ [(« - - c + rf) (N - 1) - « - - c + 3d + (2f^ - 2d)N] . 



For S > we have Pa > Pb- For large populations, N 1, we can approxi- 
mate 



< S N{a + h - c- d), 



(1.57) 



which is equivalent to 



X = 



d-h 



< 



h-c+d 2 



(1.58) 



Hence, p^ > Pb is equivalent to x* < As we have discussed above, this 
condition indicates risk dominance and establishes a relation to fixation prob- 
abilities: For weak selection, strategies with higher fixation probabilities have 
greater basins of attraction. 

Inserting Eq. \1.55\ into Eq. ( 1.16| ), we obtain an approximation for the fixa- 
tion probability of a single A individual: 



<pi 



1 + Lti nU 7/ 1 + Lti [1 - + (I + v) k)] 



.(1.59) 
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Using ZkJi ^ = ^(^ - and E£"/ = N{N - 1)(2N - l)/6, the fixa- 
tion probability becomes 

1 



1 zi; 



2N - 1 

(a-b-c + d) — a-b-c + 3d^{2b-2d)N 



r 

The same fixation probability under weak selection is found for a large variety 
of processes [54,60, 104]. For general z, we have 

i N-ii f a-b-c + d -a + bN-dN + d\ 

" N + N ( 6(iV-l) + ^) + 2(N-1) ) ■ (1-^1) 

Let us now compare the fixation probability to the result for neutral selec- 
tion, w = 0. This leads to the so called 1 / 3-rule. Neutral selection means that 
we have no selective forces and that only randomness determines fixation. In 
this case, we have (pi = 1/N. Since we are only interested if the fixation prob- 
ability is larger or smaller than 1 / N, we have to consider only the sign of F. If 
F > 0, the fixation probability is larger than 1 / N. For large N this condition 
reduces to 

a — b — c -\- d , , ^ 

^ + b - d > 0. (1.62) 

This condition is equivalent to 

'^'^ <l (1.63) 



a — b — c + d 3 

The 1 /3-rule states that "In a coordination game, the fixation probability of a 
strategy under weak selection is larger than 1 / N, if the unstable fixed point 
is closer than 1/3 to the strategy to be replaced". The intuitive justification 
of this rule goes back to the fact that during invasion, a single invader will 
interact on average 1/3 with its own type and 2/3 with the other type [79]. 

If we increase the advantage of strategy A in such a coordination game sys- 
tematically (e.g. by increasing the payoff against itself), and hence shifting the 
mixed equilibrium x* to lower values, the following scenarios occur [74]: 



X* > 2/3 A is disadvantageous and B is advantageous 

{pA < 1/N and pb > 1/N) 
2/3>x*>l/2 Bis risk dominant, but both A and B are disadvantageous 

(pA < PBr pA < 1/N and pb < 1/N) 
l/2>x*>l/3 A becomes risk dominant, but both A and B remain still 

disadvantageous {p^ > pBr pA < 1/N and pB < 1/N) 
X* < 1/3 A is advantageous and B is disadvantageous 

(Pa > 1/N and Pb < 1/N) 
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Interestingly, a analogous condition as the 1/3-rule also holds for coexis- 
tence games. In this case, the stable interior fixed point has to be closer than 
1 /3 to the strategy that is reached during fixation. In other words, "In a co- 
existence game, the fixation probability of a strategy under weak selection is 
larger than 1/N, if the stable fixed point is further than 2/3 to the strategy to 
be replaced". However, fixation probabilities are only of limited interest here, 
as fixation times become very large for large populations. The fixation times 
under weak selection are discussed in [2]. 

1.3.5 

The Fermi process 

The Moran process leads to simple analytical results under weak selection 
but no similar simplifications are possible for higher selection strengths. In 
contrast, a pairwise comparison process with p given by the Fermi function 
of the payoff difference (see above) admits simple analytical results for any 
intensity of selection. The transition probabilities are 

r± = 1^ 1 (164) 

J N N 1 + e^w(7TA-7rB) 

The analytical accessibility of this process is based on a very simple ratio of 
these transition probabilities, 

-y.^^=e-^(^^-^'^\ (1.65) 

In the weak selection limit, w <^ 1, we recover Eq. ( |1.52| ). Thus, the fixation 
probabilities are identical to the Moran process and the 1/3 rule remains valid. 
For arbitrary values of w, let us return to the ratio of fixation probabilities: 



N-l 

-A = -Q^.^exp 

7=1 



N-l 
7=1 



exp 



H 

2^. 



(1.66) 



Here, E is defined as in Eq. ( |1.56| ). Again we have pA > Pb for > 0. For large 
N, we find again that p^ > p^ is equivalent to x* < ^- But now, the relation 
between the fixation probabilities and risk dominance is valid for arbitrary 
intensities of selection, not only for weak selection. 

The expressions for fixation probabilities simplify, because the products 
over 7y reduce to sums that can be solved exactly. One special case is de- 
termined by frequency independence of the payoff difference a — c = b — d. 
This case has been termed "equal gains from switching'', because switching 
from strategy B to A leads to the same payoff change, irrespective of the op- 
ponents move [75]. In this special case, even the outer sum in Eq. ( |1.20| can be 
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solved exactly for any w. We find 

This result is identical to the fixation probability of k individuals with fixed 
relative fitness r = [17,24]. Thus, a game with equal gains from switching 
has the same fixation properties as constant selection in frequency indepen- 
dent settings with fixed fitness values. Since the Fermi process only depends 
on payoff differences, this is no surprise. But it also shows that properties of 
constant selection apply not only for the Moran process under weak selection 
but for other processes as well. 

For general payoffs, we can approximate the outer sum in Eq. ( |1.20| over k 
by an integral, E^=i • • • ~ j[. . .dk and arrive at [107] 
_ erf[Q,]-erf[Qo] 

- erf[Q^]-erf[Qo]- ^^''^^ 



Here, erf (x) = dy e is the error function and Qk — y ^^^u (ku + v) 

with u,v as in Eq. ( |1.54| ) [107]. The result is valid for u ^ and in the limit 
of 1/ ^ 0, it reduces to Eq. ( |1.67| . In the weak selection limit, zv ^ 0, Eqs. 
\1.6S) and { 1.67) recover the neutral selection result (pj^ = k/N. Numerical 



simulations of the fixation probabilities agree very well with this approxima- 
tion and hold even for small populations where the approximation of sums 
by integrals becomes inadequate (see Fig. |1.2[ ). 

The Fermi process covers all intensities of selection and leads to strong se- 
lection results that are outside the realm of the standard Moran process. The 
closed expressions allow to derive approximations for the fixation probabil- 
ities under weak and under strong selection. As for the Moran process, the 
time to fixation grows exponentially with N for games with internal Nash 
equilibria, such that fixation will practically never occur. Here, it also grows 
exponentially with the intensity of selection w. 



1.4 

From finite to infinite populations (and back again) 

So far, we have introduced two descriptions for evolutionary game dynam- 
ics: The deterministic replicator dynamics for large populations and stochastic 
evolutionary game dynamics in finite populations. Here, we discuss how both 
dynamics are related to each other. In the limit of large N, this is accomplished 
by performing a Kramers-Moyal expansion of the Master equation [30,57]: 

-Pm)Tr-Pm)T+ (1.69) 
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Fig. 1.2 Fixation probabilities for a coordination game with bistability (diagram on top). For 
neutral selection, w = 0, the fixation probabilities are a linear function of the initial number 
of A individuals (solid line). With increasing intensity of selection w, the fixation probabilities 
of the pairwise comparison process (with p given by the Fermi function) converge to a step 
function with a discontinuity at the unstable fixed point, x* = 37.5. Symbols represent the 
exact expression and lines the continuous approximation Eq. < |1.68) (payoff matrix given in the 
figure, N = 50). 



where PJ denotes the probability to be in state ; at time r. To consider large 
N, we introduce the notation x = j /N, t = r/N and the probability density 
p{x,t) = NP^(;). For the transition probabilities, we replace T^{x). 
This yields 

- p{x,t)T-{x)-p{x,t)T^{x). (1.70) 

For N 1, the probability densities and the transition probabilities are ex- 
panded in a Taylor series at x and t. More specific, we have 

/ ^ \ 3 1 

p[x,t + -^ « p{x,t) + -p(x,t)- (1.71) 

p(x±l,*) « P(-^0±^M-.0^ + |^Mx,0^ (1.72) 
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and 

r±(x± i) . r^M ± AT-(.)i + ^r-M^. (i.73) 

Let us now look at the terms depending on their order in 1/N. The terms 
independent oil/N cancel on both sides of Eq. 7Q[ ). The first non-vanishing 
term is of order 1/N. On the left hand side, we have the term (x, t) and on 
the right hand side, we have 

-p{x, t) ^ r+ (x) + p{x, t) ^ T- (x) - T+ (x) ^p{x, t) + T- (x) ^p(x, t) 

^-^[T+{x)-T-{x)]p{x,t). (1.74) 

This term describes the average motion of the system. In physics, it is called 
the drift term but in biology, it is referred to as selection term. Next, we con- 
sider terms of the order On the right hand side, we have 

+ il'^''^^) + Ip^ilT-i^) + lT-ix)^^pix,t) 

-l^[T^i^) + T-{x)]p{x,t). (1.75) 

This second term, called diffusion in physics, leads to a widening of the prob- 
ability distribution in the course of time. In biology, it is called genetic or 
neutral drift, which can be a source of confusion. In the following, higher 
order terms will be neglected. Thus, we can approximate Eq. { 1.70) by 



3^p(x,f) = -- [T^x)-T-(x)] pixj) + ^^ p(xj) 

(1.76) 

This is the Fokker-Planck equation of the system, which describes the deter- 
ministic time evolution of a probability distribution. 

Equivalently, one can describe the process by a stochastic differential equa- 
tion that generates a single trajectory. If the noise is microscopically uncorre- 
cted, as in our case, the Ito calculus has to applied [30]. In this framework, 
the Fokker-Planck equation above corresponds to the stochastic differential 
equation 



(1.77) 
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where f is uncorrelated Gaussian noise, a{x) is the drift term (selection in biol- 
ogy). In general, the diffusion term b{x) (genetic drift in biology) depends not 
only on the composition of the population, but also on the payoffs. However, 
for many processes, the payoff dependence vanishes. In particular for weak 
selection, b{x) is independent of the payoffs. Note that the noise is multiplica- 
tive and that the drift term vanishes at the boundaries x = and x = 1, which 
is important to avoid that these boundaries are crossed. For additive noise, 
processes leading to x < or x > 1 have to be excluded artificially [110] 

For N ^ 00, we have b{x) and only the term a{x) determines the 
dynamics. This case reduces to the deterministic differential equation, 

x = T^{x)-T-{x) (1.78) 



and recovers the replicator equation [104] (see Fig. |1.3| ). The same procedure 
also works for more than two strategies, although in this case, the mathemat- 
ics is more tedious [105]. A similar procedure can also be applied to spa- 
tially extended systems, where a stochastic partial differential equation is ob- 
tained [83,84]. Note that we have only considered the limit N ^ oo, keeping 
everything else fixed. To perform a thermodynamical limit, the intensity of 
selection zv has to be scaled with N [9]. 

The approximation of the process for large N can be used to address the 
evolutionary dynamics in large populations under weak selection. For exam- 
ple, we can verify the 1/3-rule. Starting from the Fokker-Planck equation, 
the probability of fixation can be written in terms of the drift and diffusion 
coefficients as 



S(j) rU^ r r 



fy aiz) , 
-az 



dy, (1.79) 



see [24,30,108]. For neutral selection, we have a[z) =0 and thus (pj = ^. 
In general, the comparison of the full fixation probability to neutral selection 
has to be done numerically. In the limit of weak selection, <C 1 the 1/3- 
rule is recovered when comparing the fixation probability of a small group of 
mutants to neutral selection [108]. More precisely, the 1/3 rule is obtained for 
Nw <^ 1. For Nw ^ 1, the result expected from the replicator equation is 
recovered: A disadvantageous rare strategy will never reach fixation with a 
significant probability. 



1.5 

Applications 



Complementing the general theory of evolutionary game dynamics we now 
turn to the two most important applications in evolutionary biology and 
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Fig. 1.3 With increasing population size, the dynamics converges to the replicator equation 
in a game where strategy A dominates B (diagram on top). We start with 20% individuals of 
type A and depict the strategy distribution after T = N,2N,3N and 4N time steps for a small 
population of size N = 100 (left) as well as a large population with N = 10000 (right). Selec- 
tion shifts the distribution towards higher fractions of A players, while the stochasticity of the 
evolutionary process widens the distribution. For small populations this quickly increases the 
variance, whereas for large populations the effect is barely visible and the distribution remains 
narrow. Parameters: payoff matrix a = 2, b = 5, c = 1, d = 3, simulations with pairwise 
comparison based on the Fermi function with w = 1, averages over 10^ realizations. 



across behavioral sciences: the problem of cooperation and the maintenance of 
biodiversity. In game theory, cooperation refers to behavioral actions that ben- 
efit others at some cost to the actor. Thus, groups of cooperating individuals 
fare better than groups of non-cooperating defectors. However, each individ- 
ual faces the temptation to defect and free-ride on the benefits produced by 
others. This generates a conflict of interest between the individuals and the 
group, which characterizes social dilemmas [18,21,43]. Social dilemmas are 
abundant in nature. They represent a recurrent theme ranging from defense 
formations in musk oxen to defend their young against wolves [36], sentinel 
behavior in meerkats [13], predator inspection in fish [65, 81], grooming in 
baboons [89,96], protein production in phages [113,114] to microorganisms 
producing extra-cellular products such as enzymes in yeast [33], biofilms [82] 
or antibiotic resistance [70], to name only a few prominent examples. How- 
ever, social dilemmas also occurred on evolutionary scales and life could not 
have unfolded without the repeated incorporation of lower level units into 
higher levels entities. Every resolution of a social dilemma marks a major 
transition in evolutionary history: the formation of chromosomes out of repli- 
cating DNA, the transition from unicellular to multicellular organisms or from 
individuals to societies all require cooperation [63]. In human interactions, so- 
cial dilemmas are equally abundant in terms of social security, health care and 
pension plans but even more importantly when it comes to the preservation of 
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natural resources from local to global scales, including drinking water, clean 
air, fisheries and climate [37,66,67]. 

The viability of ecological systems is determined by the biodiversity, which 
includes species, habitat and genetic diversity [48, 97, 103]. Species co- 
existence is promoted by non-hierarchical, cyclic interactions where R beats S 
beats P beats R, just as in the children's game Rock-Scissors Paper. This oc- 
curs in bacterial strains of E. coli, where a neutral strain is eliminated by a toxin 
producing strain, which is then outgrown by an immune but non-toxic strain, 
which is in turn outgrown by the neutral strain and so on. Further examples 
of cyclic dominance hierarchies include mating strategies in lizards [94, 95] or 
competition for space in coral reef invertebrates [55] and links to the prob- 
lem of cooperation if participation in social dilemmas is voluntary rather than 
compulsory [39,40,92] (see Sect.[L53|. 



1.5.1 

The Prisoner's Dilemma 

Let us begin with the Prisoner's Dilemma, which has a long tradition as a 
mathematical metaphor to analyze the problem of cooperation [4,5]. In the 
Prisoner's Dilemma, two players can choose between cooperation and defec- 
tion. A cooperative act costs c > 0, but leads to a benefit b > c for the other 
player. Thus, the highest payoff, h, is obtained when only the partner is coop- 
erating. In this case, the partner obtains — c. Mutual cooperation leads ioh — c 
and mutual defection a zero payoff. The game is characterized by the payoff 
matrix 



C D 

C f h — c —c 
D\h 



(1.80) 



The Prisoner's Dilemma represents the most stringent form of a social dilemma 
because the strategy D dominates strategy C. No matter what the opponent 
does, one should defect since b > b — c and > — c. Unilateral deviation from 
mutual defection decreases the payoff and hence mutual defection represents 
the only Nash equilibrium [51,69] but mutual cooperation corresponds to the 
social optimum (b — c > 0). 

Note that the parametrization in terms of costs and benefits represents 
the most intuitive and mathematically convenient form of the Prisoner's 
Dilemma. However, it is equally important to note that this reflects a spe- 
cial case because the sum of the diagonal elements equals the sum of the 
non-diagonal elements of the payoff matrix. In other words, the game is an ex- 
ample of "equal gains from switching" (see section 1.3.5[ ). This property leads 



to a payoff difference between cooperators and defectors, ttq — TCiy = — c, 
that is independent of the fraction of cooperators xq- In this special case the 
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replicator dynamics reads 
xc = -xc{l-xc)c (1.81) 

and can be solved exactly: xc{t) = xc(0) [xc(0) + (1 — xc{0)) e^^^] ^ . The 
fraction of cooperators xq is always decreasing and converges to the only sta- 
ble fixed point xq = 0. Cooperators are doomed and disappear. 

In finite populations and under weak selection, <C 1, we find, in agree- 
ment with Eq. ( |1.61| ), for the fixation probability of i cooperators in a popula- 
tion oi N — i defectors 

i i N — i f b \ Nw i 

Since (pi < j^, cooperators are at a disadvantage compared to neutral mutants. 
Similarly, for strong selection, we find from the Fermi process in the limit w 
00 the fixation probabilities (pi = Si^j^. In other words, cooperation cannot 
evolve from individual selection alone. 

The stark contrast between theoretical predictions of the Prisoner's Dilemma 
and the observed abundance of cooperation in nature calls for explanations. 
Over the last decades, a number of mechanisms capable of promoting cooper- 
ation in biological and social systems have been proposed [21, 72]. Among 
related individuals, cooperators may thrive due to kin selection [35] and 
competition among groups may promote cooperation through group selec- 
tion [26, 106, 116, 117]. Conditional behavioral rules that strategically respond 
to previous encounters in repeated interactions or adjust their behavior ac- 
cording to an individuals' reputation in non-repeated settings can establish co- 
operation through direct [112] or indirect reciprocity [76]. Local interactions in 
structured populations support cooperation in the Prisoner's Dilemma [73,80] 
but not necessarily in other social dilemmas [41]. Finally, extensions of the 
strategy space that enable individuals to abstain from social dilemmas and 
render participation voluntary or options to punish non-cooperating interac- 
tion partners both support cooperation [40,45] (see Sects. 1.5.3 , L5^. 



1.5.2 

Rock-Paper-Scissors 

Rock-paper-scissors games are the simplest example of cyclic dominance, 
where any strategy can be beaten by another one: Rock crushes scissors, scis- 
sors cut paper and paper wraps rock. This simple game not only entertains 
children (and adults [52]) but equally serves as a mathematical metaphor to 
investigate the dynamics and maintenance of biodiversity [83,99,100]. The 




Fig. 1.4 Depending on the determinant of the payoff matrix d, two qualitatively different sce- 
narios of cyclic dominance are possible in the game of Rock-Paper-Scissors, (a) If the deter- 
minant d is positive, the interior fixed point is asymptotically stable and all orbits converge to 
this point, (b) If the determinant d is negative, the interior fixed point is unstable and the sys- 
tem approaches a heteroclinic cycle along the boundary of the simplex (the figure is produced 
with Bill Sandholm's Dynamo package [86]). 
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(1.83) 



This parametrization assumes symmetric strategies but the payoff for winning 
(+1) is not necessarily equal to the payoff for losing (— s < 0). For the standard 
choice s = 1, we have a zero-sum game - one player's gain is the other ones' 
loss. 

Apart from the three trivial homogeneous equilibria, the replicator dynam- 
ics admits a non-trivial equilibrium at x* = {x^,x^,x^^ = (\,\,\)' The dy- 
namics of the system is determined by the Lyapunov function 

H = -Xi^xpxs. (1.84) 



If the determinant of the payoff matrix dL83|), d = 1 — s^, is positive, then 



^ < and the interior fixed point x* is asymptotically stable. For d < 0, the 
fixed point x* is unstable and the system approaches a heteroclinic cycle along 
the boundary of the simplex S3. Finally, for the zero-sum game with s = 1, 
the function ( |1.84| ) is a constant of motion, and the system infinitely oscillates 



around x*, see Fig. 1.4 
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In finite populations, the dynamics is stochastic but characteristic features 
can be derived from the average drift towards the internal fixed point [11]. 
For large N, the average drift of the Moran process computed over the entire 
simplex is given by 



For N ^ 00, the replicator equations lead to AH = for zero-sum games 
(s = 1) as well as for neutral selection (k; = 0). In finite populations, we obtain 
(AH)]v = > in both cases instead. For s = 1, stochasticity turns the 
neutrally stable, closed orbits of infinite populations into an unstable system 
by introducing a drift towards the heteroclinic cycle. Only in the limit N — > oo, 
the neutrally stable orbits characterized by AH = are recovered from the 
Moran process or other forms of finite population dynamics. However, even 
in the infinite system it is not trivial to numerically determine these closed 
orbits of the replicator equation and the numerical integrator has to be chosen 
with great care [49]. 

For s < 1, the sign of (AH)]v depends on the intensity of selection w, the 
payoff parameter s as well as on the population size N. This leads to a critical 
population size 



Since 42 is the answer to life, the universe, and everything, this result is not 
surprising [1]. For N < Nc, the system cycles towards the boundaries and 
fixation is expected to be fast. For N > N^, the system converges towards 
on average. 

This game is an example of a system that changes its qualitative dynamics 
if the population falls below a certain threshold. Such thresholds are often 
found in other evolutionary games in finite populations [10, 74, 101, 104]. 




(1.85) 
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1.5.3 

Voluntary Public Goods Games 

So far, we focussed on interactions among pairs of individuals. Now, we turn 
to Public Goods Games [56] in which a group of M players interact. Public 
Goods Games represent a generalization of pairwise Prisoner's Dilemma in- 
teractions to interaction groups of arbitrary size M [44]. 

In typical Public Goods experiments, M individuals have the opportunity 
to cooperate and invest a fixed amount c into a common pool or to defect and 
invest nothing. The total investment is multiplied by a factor r and distributed 
equally among all participants - irrespective of whether they contributed or 
not. Thus, every invested unit returns rc/M units to the investor (as well as 
to all other participants). If r < M then rational players withhold their invest- 
ments because they are costly - but if all participants reason in this manner, 
no investments are made and the group foregoes the benefits of the public 
good. In contrast, had everybody cooperated, they would have been better 
off with a return of (r — l)c. Again we have a social dilemma and the Nash 
equilibrium is different from the social optimum. However, if r > M then 
investments have a positive net return, and rational players will invest in the 
public good [43]. 

In infinite populations with a fraction x cooperators and y defectors (x + y = 
1) the average payoff of defectors is rtjj = (M — l)xrc/ M and of cooperators it 
is 7tc = 7Zd — (1 — r/ M)c. Thus, for r < M cooperators decline and eventually 
disappear because ttq < tzd. However, for r > M cooperators dominate 
(ttc > tcd) and eventually displace defectors. In this case cooperation evolves 
as a by-product [15]. 

The above analysis is based on the assumption of compulsory participation 
in public goods interactions but what happens if participation is voluntary 
and risk averse individuals may choose to abstain? This situation can be mod- 
eled by introducing a third strategic type, the non-participating loners [39,40]. 
Loners obtain a fixed payoff a, which lies between the payoff in groups con- 
sisting solely of cooperators, (r — l)c, and the payoff in groups of defectors 
(which is zero). Interaction groups are randomly formed by sampling M indi- 
viduals from an infinite population with a fraction x cooperators, y defectors 
and z loners. Thus, the effective interaction group size S of the Public Goods 
Game decreases with increasing abundance of loners. If only a single individ- 
ual participates in the Public Goods interaction, the game does not take place 
and the individual is forced to act as a loner. The average payoffs of defectors. 
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cooperators and loners are given by [39]: 



Tic, = az^-^ + r c—^ (1 



Ttc = ttd - 1 + (r - l)z 



1-z V M(l-z) 

r 1-z^ 



M 

TtL = a (1.87) 

Note that with few loners, z ^ 0, defectors dominate cooperators, tid > ^C/ 
but for z ^ 1, cooperators outperform defectors, tcq > tcid. This generates 
a Rock-Scissors-Paper type cyclic dominance among the three strategies: if 
cooperators abound it pays to defect and if defection is common it is best to 
abstain but this reduces the effective interaction group size S until eventually 
r > S holds and cooperators thrive, which in turn increases S, restores the 
social dilemma and the cycle continues, see Fig. |1.5{ 3. By definition S > 2 must 
hold because a single participant (S = 1) is unable to sustain the public good. 
Therefore, the above reasoning requires r > 2. In this case, a heteroclinic orbit 
along the boundary of the simplex S3 reflects the cyclic dominance of the three 
strategies and in the interior a neutrally stable fixed point Q exists, which 
is surrounded by closed orbits such that the system exhibits stable periodic 
oscillations of the strategy frequencies, see Fig. |1.5{ 3. For r < 2, the heteroclinic 
orbit still exists but the interior is filled with homoclinic orbits of the state with 
all loners, z = 1. Therefore, only brief intermittent bursts of cooperation are 



observed before the system returns to the loner state, see Fig. 1.5 

In order to allow for a compact analytical description of the stochastic dy- 
namics in finite populations, we assume small mutation rates }i such that 
a strategy reaches fixation or extinction much faster than mutants are pro- 
duced [29,45,53]. This is fulfilled if }i is much smaller than the inverse of the 
squared population size. Thus, the system is usually homogeneous and only 
occasionally switches from one homogeneous state to another. The dynam- 
ics is essentially reduced to a stochastic process along the boundaries of the 
discretized simplex S3 and we can approximate the system by an embedded 
Markov chain on the pure states of the system, C, D and L. The average time 
the system spends in each state depends on the updating process, the intensity 
of selection as well as on the game parameters. For simplicity, let us consider 
imitation dynamics (c.f. Eq. ( |1.6[ ) in the limit of strong selection, z(; ^ 00). In 
this case, a single defector takes over a cooperator population with probabil- 
ity 1. Similarly, the probability that a single loner takes over a population of 
defectors is also 1. Finally, a loner population is taken over by cooperators 
with probability 1/2. This is because the first cooperator is neutral (no Public 
Goods games take place) and disappears with the same probability as it gets 
imitated. However, as soon as there are two cooperators, they have an advan- 
tage and eliminate the loners with certainty. This leads to the transition matrix 
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Cooperators 



b 



Cooperators 



a 




Loners 



Defectors 



Loners 



Defectors 



Fig. 1.5 Dynamics in voluntary Public Goods Games in which cooperators are dominated by 
defectors, defectors by loners and loners by cooperators. (a) For r < 2, only brief intermittent 
bursts of cooperation are observed before the system settles again in the loner state, (b) For 
r > 2, a neutrally stable fixed point Q appears, which is surrounded by closed orbits. The 
fraction of cooperators performs stable oscillations (parameters: (a) c = 1, r = 1.8 and cr = 
0.5. (b) c = 1, r = 3 and (7 = 1). 



among the three homogeneous states: 



Note that the system stays e.g. in state C with probability 1/2 because the 
probability that the mutant is an unsuccessful loner is 1/2. Under imitation 
dynamics the transition matrix is parameter independent as long as the cyclic 
dominance applies (which only requires r < M and < a < r — 1). The sta- 
tionary distribution P is given by the eigenvector corresponding to the eigen- 
value 1 of this stochastic matrix, P = (PcrPDrPi) = (5/5/2)- Thus, the system 
spends 50% of the time in the loner state and 25% in the cooperator and and 
defector states, respectively. In compulsory interactions, i.e. in the absence of 
loners, the system would spend essentially all the time in the defector state. 
Interactive Java simulations of this system can be found online [38]. 

In summary, voluntary participation in Public Goods Games provides an 
escape hatch out of states of mutual defection. This maintains cooperation, 
but fails to stabilize it. The cyclic dynamics of voluntary Public Goods Games 
has been confirmed in behavioral experiments [92]. In the next section, we 
demonstrate that the time spent in cooperative states can be vastly extended 
by introducing opportunities to punish defectors. 



C 



D 



L 




(1.88) 
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1.5.4 

Punishment 

Punishment is ubiquitous in nature ranging from bacterial colonies to human 
societies [12,23,25,27,34,93]. In game theoretical terms, punishment is de- 
fined as behavioral actions that impose a fine j6 on a co-player at some cost 
7 < j6 to the actor. Punishment enables cooperators to retaliate against non- 
cooperating defectors and therefore seems capable of stabilizing cooperation. 
However, this raises a second-order social dilemma because non-punishing 
cooperators outperform those that do punish. Thus, in an evolving popula- 
tion, mild cooperators undermine the punishers' efforts and pave the way for 
the successful invasion of defectors. Moreover, if punishers are rare, they suf- 
fer tremendous costs from punishing left and right, while inflicting little harm 
on the defecting members of the population and hence it remains unclear how 
punishment behavior could have gained a foothold in a population. Colman 
aptly summarizes this by stating: "we seem to have replaced the problem of 
explaining cooperation with that of explaining altruistic punishment" [14]. 

In the previous section we demonstrated that voluntary participation in 
Public Goods Games promotes cooperation without being able to stabilize it. 
Here we extend this setting by introducing a fourth strategic type, the pun- 
ishers. Punishers are individuals that cooperate but, in addition, punish de- 
fectors. The combination of volunteering an punishment was originally pro- 
posed by Fowler [28], but it turns out that the replicator dynamics in infinite 
populations is unsuitable to draw clear-cut conclusions because the system is 
bi-stable (i.e. the evolutionary end state depends on the initial configuration) 
and also structurally unstable (i.e. the evolutionary dynamics may undergo 
significant changes upon introducing stochastic components) [6]. 

In the following, we consider the stochastic dynamics of volunteering and 
punishment in Public Goods Games in finite populations [45,46]. As in the 
case without punishment, we take the limit of small mutation rates and strong 
selection. Again, this implies that the population is homogeneous most of the 
time, i.e. in states C, D, L or P. An occasional mutant will have taken over 
the population or disappeared before the next mutation arises. Therefore, the 
dynamics is determined by a Markov chain based on the different transition 
probabilities. In the limit of strong selection, the derivation of these transition 
probabilities is particularly simple: (i) in the cooperator state C, a single defec- 
tor is advantageous and takes over the population with probability 1. Loners 
are disadvantageous and cannot invade. Punishers can invade through neu- 
tral drift with probability . (ii) in the defector state D, a mutant cooperator 
or punisher is disadvantageous and disappears. Loners are advantageous and 
take over with probability 1. (iii) in the loner state L, the djmamics is neutral 
if a mutant of any type arises. With probability 1/2 another individual adopts 
the mutant strategy (or the mutant disappears with the same probability). A 
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pair of mutant cooperators or punishers is advantageous and takes over but 
a pair of defectors is disadvantageous and at least one disappears, (iv) in the 
punisher state P, both defectors and loners are disadvantageous. The former 
because they are punished and the latter because they do not take advantage 
of the common good. However, cooperators obtain the same payoff as pun- 
ishers and can take over through neutral drift with probability ^. This yields 
the transition matrix 



C 
D 
L 
P 



C D L P 

/ 3 3N ^ 6 3N 
1 2 
3 3 

Olio 







\ 3N ^ 6 ^ 3N / 



(1.89) 



which is again independent of the interaction parameters (c.f. Eq. ( |1.88[ )) and 
only depends on the population size N because of the neutral transitions be- 
tween cooperators and punishers. The stationary distribution becomes 



(1.90) 



-N 



Thus, for large N, the system spends almost all the time in the punisher state. 
The reason is that the transition leading away from the punisher state is neu- 
tral and thus very slow compared to all other transitions. The punisher state is 
the only pure Nash equilibrium, but it is not a strict Nash equilibrium, as co- 
operators are equally well off. Despite the vanishing time the system spends 
in the loners' state, voluntary participation plays a pivotal role for cooperation 
and punishment because it provides recurrent opportunities for establishing 



social norms based on punishing non-cooperating defectors, see Fig. 1.6 

In contrast, in compulsory Public Goods Games, i.e. in the absence of lon- 
ers, the cyclic dominance of C, D, and L is missing and once cooperation 
breaks down, it cannot get re-established, see Fig. |1.6[ Interestingly, punish- 
ment emerges only if the participation in Public Goods Games is voluntary. 
This conclusion nicely relates to recent experiments, where individuals can 
choose whether to join a Public Goods Game with or without punishment, 
they voluntarily commit themselves to the sanctioning rules [34]. For interac- 
tive simulations of volunteering and punishment, see [38]. 

This approach to punishment, which is most common in behavioral eco- 
nomics, raises moral concerns because enforcing cooperation through peer- 
punishment means that individuals take the law into their own hands, but 
mob law is clearly socially unacceptable. 
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Fig. 1.6 For small mutation rates and strong selection, the stochastic dynamics of finite popu- 
lations can be approximated by an embedded Markov chain on the pure states of the system, 
(a) For the voluntary Public Goods Game, a cyclic dominance between cooperators, defectors 
and loners emerges. We find that the system spends 50% in the Loner state and 25 % in the 
cooperator and defector states, respectively. This result is independent of all parameters, (b) 
In voluntary Public Goods Games with punishment, punishers can invade the loner state. The 
system then spends a long time in the punisher state because only cooperators can invade 
and only through neutral drift. Once the system is in the cooperator state it is prone to invasion 
by defectors and may undergo several cooperator-defector-loner-cycles before punishment 
is re-established. In the long run, punishers dominate the population most of the time, (c) In 
compulsory Public Goods Games with punishment, the ultimate outcome is defection. Even 
when starting in the punisher state, cooperators can invade by neutral drift and once defectors 
take over, the system has reached its evolutionary end state (population size N = 92). 



1.6 

Concluding remarks 

This review mainly focusses on the particularly insightful class of 2 x 2 games. 
We have analyzed the stochastic dynamics of evolutionary games in finite 
populations, as it is, for example, described by the Moran process. The connec- 
tion to the traditional description of evolutionary games by the deterministic 
replicator equation is established through approximations in the limit of large 
populations. 

As applications and extensions of the theoretical framework, we provide 
brief excursions into interactions with more strategic options, as in the case 
of Rock-Scissors-Paper games, which are relevant in the context of biodi- 
versity, as well as to interactions among larger groups of individuals to ad- 
dress the problem of cooperation in the Prisoner's Dilemma and in Public 
Goods Games with voluntary participation and /or punishment opportuni- 
ties. Other important games that have not been covered here include the 
Snowdrift Game [22,41], where cooperation is not strictly dominated by de- 
fection, and the Minority Game, which turned into a paradigm for simplified 
market models [7,8,16]. 
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Further important directions of the field include spatial games, which have 
recently been summarized in an excellent review by Szabo and Path [98], as 
well as recent advances in ecological games with variable population densi- 
ties [42, 47] and for games with continuous strategy spaces [22, 59] based on 
adaptive djmamics [19,31,64]. 

Over the past few years, the tremendous progress of our understanding of 
evolutionary dynamics can be largely attributed to numerous insights gained 
from stochastic djmamics in finite populations as well as from considering the 
analytically accessible limits of weak selection and rare mutations. The often 
counter intuitive results are inaccessible from traditional approaches based on 
replicator equations. 

Stochastic evolutionary game djmamics is more difficult to handle analyti- 
cally as compared to deterministic approaches such as the replicator dynam- 
ics. Nonetheless, it is a very powerful tool, because it implements a natural 
source of noise that actually renders the results more robust. For example, 
in bistable systems it allows to calculate transition rates between the different 
states rather than concluding that the dynamics depends on the initial con- 
ditions. Most importantly, any real population is subject to noise and incor- 
porating such essential features into game theoretical scenarios significantly 
improves their relevance in modeling the real world. 
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